Papers with data augmentation baselines
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that the trillion parameter mark for pre-trained language models is not achievable without a change in training paradigm. |
| Approach: | They propose a text-based adversarial training algorithm which enhances the performance of knowledge distillation by maximizing the divergence between teacher and student logits. |
| Outcome: | The proposed algorithm outperforms competing adversarial learning and data augmentation baselines on the GLUE benchmark. |
COCOGEC: Counterfactual Generation for Robust Grammatical Error Correction (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing GEC models fail to understand error patterns in varying contexts . a framework that generates copies of training instances with error-irrelevant contexts altered is proposed . |
| Approach: | They propose a framework that generates copies of training instances with error-irrelevant contexts altered. |
| Outcome: | The proposed framework outperforms baselines on the simulated tasks and outperformed existing models. |
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024.findings-acl)
Copied to clipboard
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Pretrained large language models are currently state-of-the-art for solving most tasks . however, many of them are in the low-data regime, making fine-tuning challenging . a new data augmentation strategy uses a teacher LLM to augment a small seed dataset . |
| Approach: | They propose a targeted and iterative data augmentation strategy that augments a teacher LLM to fine-tune a small seed dataset by adding additional data. |
| Outcome: | The proposed approach outperforms fine-tuning and other data augmentation strategies on a small seed dataset. |
Simple Conversational Data Augmentation for Semi-supervised Abstractive Dialogue Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive conversation summarization models heavily rely on human-annotated summaries. |
| Approach: | They propose a set of Conversational Data Augmentation methods for semi-supervised abstractive conversation summarization that use random swapping/deletion to perturb the discourse relations inside conversations and dialogue-acts-guided insertion to interrupt the development of conversations. |
| Outcome: | The proposed methods over several state-of-the-art datasets show that they are more efficient than previous methods. |